acc: run ssh/connect-serverless-gpu locally#5878
Merged
Merged
Conversation
Flip acceptance/ssh/connect-serverless-gpu to Local = true so it runs against the in-process testserver (it was disabled on both cloud and local in #4838). A full SSH handshake can't complete locally (no real compute/sshd + a wss driver-proxy), so the local run drives the connect flow up to the connection attempt and asserts the serverless GPU bootstrap job the CLI submits (accelerator, environment, secret scope). Teach libs/testserver enough of the SSH tunnel backend for the flow to get there deterministically and fast: - secrets/list (ListSecretsByScope), returning RESOURCE_DOES_NOT_EXIST for a missing scope so CreateKeysSecretScope creates it - synthesize metadata.json on the ssh-server-bootstrap run submit, as a real tunnel server would publish it, so getServerMetadata succeeds - driver-proxy metadata/logs endpoints - workspace/export now 404s a missing object instead of returning a nil body (which the response normalizer rejects) The connect attempt itself is expected to fail locally; its noisy output goes to LOG.stderr (excluded from the golden files).
Collaborator
Integration test reportCommit: 102a096
10 interesting tests: 4 SKIP, 3 flaky, 3 RECOVERED
Top 9 slowest tests (at least 2 minutes):
|
Add a proxy-level test that runs a real OpenSSH server (sshd -i) behind the websocket proxy and a real SSH client through RunClientProxy, exercising the actual handshake, pubkey auth, and remote command exec end-to-end without any Databricks compute. This complements the existing `cat`-based transport tests by validating that the tunnel carries the SSH protocol faithfully, giving us local coverage of the connectivity path that otherwise needs a cluster. The test is skipped when sshd (openssh-server) is not installed and is gated to non-Windows, matching the existing proxy tests.
The runner images ship only openssh-client, so the real SSH handshake test would silently skip on Linux (no sshd) and is unreliable to run as a non-root user on macOS. Restrict it to Linux and install openssh-server in the test-exp-ssh job so it runs deterministically there; it still skips gracefully when sshd is absent (e.g. the general test job, which doesn't install it).
anton-107
approved these changes
Jul 13, 2026
anton-107
left a comment
Contributor
There was a problem hiding this comment.
LGTM, but let's try and converge these tests in two different styles into one (acceptance) in one of the follow-up PRs
radakam
added a commit
that referenced
this pull request
Jul 14, 2026
Apply the same convergence to the serverless-GPU acceptance test (follow-up to #5878, where the review asked to fold the two test styles into one acceptance test). Locally it now runs a single `ssh connect --proxy` that submits the same serverless-GPU bootstrap job and pipes stdin/stdout over the driver-proxy websocket echo route, asserting both the job payload and the echoed bytes.
ronaldz-db
pushed a commit
to ronaldz-db/cli
that referenced
this pull request
Jul 15, 2026
## Changes The dedicated-cluster counterpart to `acceptance/ssh/connect-serverless-gpu` (databricks#5878). Flips `acceptance/ssh/connection` to `Local = true` and asserts the bootstrap job the CLI submits for `ssh connect --cluster` (`existing_cluster_id`, secret scope, notebook task, `serverless=false`), reusing the testserver SSH-tunnel stubs from databricks#5878. The default test cluster (`TestDefaultClusterId`) is now a running dedicated single-user cluster — the shape `ssh connect --cluster` requires (`ValidateClusterAccess` rejects anything else), matching the cloud `TEST_DEFAULT_CLUSTER_ID` it stands in for. Only `bundle/deployment/bind/cluster` (filters to `cluster_name`) and the `cmd/api` route stubs reference it, so no other goldens change. ## Tests - `acceptance/ssh/connection` (asserts the submitted dedicated-cluster bootstrap job payload)
radakam
added a commit
that referenced
this pull request
Jul 15, 2026
#5920) ## Changes Follow-up to #5878, converging the SSH tunnel's two test styles into one acceptance test. Two enabling changes: derive the driver-proxy websocket scheme from the host (`http` → `ws`, else `wss`) instead of hardcoding `wss` so the client can dial the plaintext local test server, and add a testserver `/ssh` websocket route (via a new connection-hijacking `Router.HandleRaw`) that upgrades the driver-proxy request to a **real `sshd -i`** and bridges binary frames to its stdio, standing in for the tunnel server a real cluster runs — no Databricks compute required. The `/ssh` handler reads the client public key the CLI uploaded via `secrets/put` (the ws dial carries the bearer token, so the token-scoped workspace is the one that stored it), writes an ephemeral host key + `authorized_keys` + a minimal `sshd_config`, and pumps websocket frames to and from `sshd`'s stdin/stdout. `/metadata` now returns the current OS user so `ssh -l <user>` matches the account `sshd` runs as. `acceptance/ssh/connect-serverless-gpu` now runs a single local `ssh connect ... -- "echo 'Connection successful'"` that both submits the serverless-GPU bootstrap job and completes a genuine SSH handshake, public-key auth, and remote command exec over the `ws://` tunnel, asserting both the job payload and the command output. This makes `TestClientServerRealSSHD` redundant, so it's removed; the `cat`-echo transport tests in `client_server_test.go` remain. Note the converged test exercises the real handshake on the **client side only**: the `/ssh` route is a testserver stand-in, not the production `proxy.NewProxyServer`/`runServerProxy`, so the server proxy's transport stays covered by the `cat`-echo tests rather than by a real handshake. This is a deliberate trade-off (`runServerProxy` is protocol-agnostic), not an accidental coverage drop. The handshake needs a real OpenSSH server: the test self-skips via `SKIP_TEST` when `sshd` is absent (the general `test` job doesn't provision it; `task test-exp-ssh` installs `openssh-server` on Linux). It's pinned to Linux (`darwin`/`windows` disabled), where running `sshd -i` as a non-root user with a throwaway config is reliable — matching the removed Go test's rationale. ## Tests * `acceptance/ssh/connect-serverless-gpu` (bootstrap job payload + real SSH handshake and remote exec over the `ws://` tunnel) * `TestBuildProxyWebsocketURL` (`ws`/`wss` scheme selection)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Local test coverage for
databricks ssh connect, which was disabled on cloud and local in #4838, across the two layers we can exercise without compute:acceptance/ssh/connect-serverless-gpuruns against the in-process testserver and asserts the serverless-GPU job the CLI submits (GPU_1xA10, serverless env, secret scope, notebook task). This needs newlibs/testserverstubs:secrets/list→RESOURCE_DOES_NOT_EXISTfor a missing scope, a synthesizedmetadata.jsononssh-server-bootstrap-*submit, driver-proxymetadata/logs, andworkspace/export→404for a missing object.client_server_sshd_test.gostands up a realsshdbehindproxy.NewProxyServerand drives a real SSH client (x/crypto/ssh) throughRunClientProxy. Thetest-exp-sshjob installsopenssh-serveron Linux to support it.Why
ssh connectcurrently has no automated coverage. These tests exercise the two layers reachable locally: the control-plane setup (the bootstrap job the CLI submits) and the tunnel transport carrying the real SSH protocol.A full local handshake via the connect command can't complete (no compute, a
wss://driver-proxy dial, realsshspawn), so the acceptance test stops at the connection attempt — failure output goes toLOG.stderr(excluded from goldens), and the cloudConnection successfulpath is preserved for when cloud is re-enabled. The handshake test is Linux-only (//go:build linux): runningsshd -ias non-root is reliable there but not on macOS; it skips whensshdis absent. The end-to-end path through the real driver proxy (bootstrap notebook →databricks ssh serveron compute) still needs an integration test and is out of scope.Tests
acceptance/ssh/connect-serverless-gpu(asserts the submitted bootstrap job payload)TestClientServerRealSSHDinexperimental/ssh/internal/proxy(real SSH handshake, pubkey auth, and remote command over the proxy tunnel), running on Linux in thetest-exp-sshjob